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Abstract: This paper describes a mechanism developed by the authors to gather student feedback from formative 
revision Multiple Choice Questionnaires (MCQs) within an on-line learning system. The MCQs provided first year 
Computer Science students with instant formative feedback, while data was also gathered about student responses, 
such as the percentage opting for each answer and the time taken to answer the question. We measured how students 
were using our on-line learning system; whether they were in fact ‘workers’ who provided answers to the MCQs, were 
lurkers’ who did not provide answers but asked for solutions or ‘shirkers’, who did not access the site at all! 

The data indicates that the time taken to answer a harder question was less than that of an easier question suggesting 
that the workers turned into lurkers strategically when they thought they could not answer successfully. It was not 
however clear whether the lurker suddenly finding an easier question would change back into a worker. Future work to 
encourage the shirkers to participate is also discussed. 
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1. Introduction 

The emergence of Virtual Learning Environments 
(VLE) in Higher Education has led to many 
improvements in student learning experiences as 
identified by Britain and Liber, (1999) amongst 
others. However, there are pitfalls in using VLEs 
and their effect on student learning has been 
taken up elsewhere, (e.g.Stiles, 2000). One major 
pitfall of VLEs in the authors’ opinion is that of 
monitoring student access to on-line teaching 
material. In the class-room environment tutors 
frequently monitor student attendance by taking 
registers. This means that non-student attendance 
at classes is recorded. Even though most if not all 
VLEs require authentication, by use of a 
username and password, to access on-line 
teaching material, monitoring student access to 
on-line material is not a simple task. For example, 
here at the University of Hertfordshire we can 
monitor the number of student logins, but as 
lecturers we cannot yet identify who is logging in 
and using our VLE. This means we have an 
overall idea of the proportion of our VLE users to 
the proportion of non-VLE users. In this paper, we 
classify those students that do not login to our 
VLE as ‘the shirkers'. 

Many VLEs provide tutors with the option of 
providing students with formative multiple-choice- 
questionnaires (MCQs) or tasks to undertake and 
it is common for tutors to provide solutions to such 
MCQs. Clarke et al (2004) discuss the 
pedagogical use of MCQs for formative testing in 
detail. However, this feedback is one-way, from 
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the tutor to the student which we call the 
tutor —» student feedback, since there is no facility 
for feedback from the student to the tutor. 

This has led us to identify a number of problems 
which we attempt to address in this paper. 
Namely these problems are: How does the tutor 
know the proportion of students who are actually 
undertaking the formative exercises? How does 
the tutor know if a student is just looking at the 
solutions and not doing the formative exercises? 
How does a tutor know how long a student is 
spending on each formative exercise? 

In the class-room, the environment is under the 
control of the tutor and the tutor gathers feedback 
from students from a number of sources that help 
to address these problems, which we call the 
student—>tutor feedback. The tutor can observe, 
talk and time students as they tackle set exercises 
and the tutor can adapt and amend his/her 
teaching approach accordingly. This type of 
dynamic teaching control is not usually available 
through a VLE. 

We have refined our on-line learning system to 
address the three questions we have posed and 
attempt to provide the tutor with the control that 
he/she has in the class-room. The idea behind our 
learning system is to monitor in the back-ground 
what students are doing when they access our 
formative MCQs or tasks on the University’s VLE. 
Currently we are monitoring anonymously, that is 
we do not monitor a username and we cannot 
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therefore identify individual students. Our 
approach is that when a student opens a web¬ 
page to try one of our MCQs or one of our tasks 
then a timer is started and statistics gathered. 
Every MCQ or task requires some form of an 
interactive answer and this response is captured 
by our learning system. When the student submits 
a response or instead requests the solution to the 
MCQ or task the timer is stopped. In both cases, 
we provide the student with our solution to the 
MCQ or task (the tutor—>student feedback). From 
this simple information, we gathered a number of 
statistics. The statistics are broken down into two 
types: the ‘lurkers’ and the ‘workers’. The lurkers 
are those students who view our formative 
teaching material and do not respond other than 
to receive the tutor’s solution. The workers are 
those students that answer our formative MCQs. 
We use these statistics to tell us the proportion of 
students that are ‘shirkers’, ‘workers’ or ‘lurkers’. 

We gather further statistics from the workers to tell 
us the proportion of students that correctly and 
incorrectly answer each MCQ or each task. We 
also gather statistics about the time taken by each 
student to answer each MCQ or the time taken to 
undertake each task. We have now achieved the 
student—> tutor feedback, which we can act on. 
For example, if the number of students answering 
an MCQ incorrectly is higher than we expect or it 
takes the average student a longer time to answer 
the MCQ than we expected then we have 
identified an area that students are struggling to 
understand and we can further adapt our VLE 
teaching material to address the problem that has 
been highlighted through the student —»tutor 
feedback. 

The layout of the rest of this paper is as follows. In 
Section 2, we briefly describe the functionality of 
our VLE, known as StudyNet. In Section 3, we 
describe our experiences of teaching Computer 
Science on a VLE. In Section 4, we discuss our 
on-line learning system and evaluate our 
feedback results. Finally in Section 5, we provide 
conclusions and a discussion. 

2. StudyNet 

The University of Hertfordshire firmly believes that 
the use of VLEs in learning and teaching is an 
important tool for both students and staff. This led 
to the university becoming one of the first UK 
universities to use a campus-wide VLE with on¬ 
line access to modules for all students and staff 
through its own in-house VLE called StudyNet 
(Jefferies et al, 2004). 

StudyNet enables students and staff to access 
information through a web-browser both on- and 
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off-campus. This means that all students and staff 
have access from a computer system on campus 
or over the internet to their own customised web- 
space. From the student perspective, this links 
students to module descriptions, lecture notes, 
university and module news items, on-line 
discussions, and group discussions, 
assessments, the university’s library system, the 
university’s e-mail system, the Students Union, 
the Careers Service and other information 
resources. StudyNet has therefore become the 
main source of information between students, 
academics and the university itself (Thornton et al, 
2005). 

In a recent report McNab (2003) states that from a 
potential number of StudyNet users of over 
21,000 almost 16,000 (72%) of users login and 
use StudyNet. This means, from the perspective 
of our VLE, that 72% of our students are workers 
or lurkers and only 28% of our students are 
shirkers. In particular, in our Faculty of 
Engineering and Information Sciences the number 
of StudyNet users were reported by Macnab to be 
over 3,700, which is about 90% workers/lurkers 
and 10% shirkers. Surprisingly, the Faculty of 
Engineering and Information Sciences was not the 
highest proportion of users; this was achieved by 
the Faculty of Law where there are over 1,000 
users, which is about 96% workers/lurkers and 
only 4% shirkers. Perhaps not so surprisingly, the 
Faculty of Art and Design achieved the lowest 
proportion of users with over 750 users, which is 
about 59% workers/lurkers and 41% shirkers. This 
StudyNet usage range of 59% to 96% of 
workers/lurkers is extremely encouraging and is 
indicative of the emphasis that the University of 
Hertfordshire has placed on the accessibility of 
on-line learning through StudyNet 

3. Student use of on-line revision 
MCQs 

In this study we targeted our MCQs towards first 
year undergraduate Computer Science students. 
We chose first year students for two main 
reasons. First, because we considered that first 
year students are more likely to be receptive to 
using our VLE and we wished to capture a true 
reflection of the ratio of workers, lurkers and 
shirkers. Second, at least two of our first year 
taught modules use MCQs as summative 
assessment to compose student grades. 

3.1 Study 1 

In our first study, we decided to invite our students 
to undertake 50 formative revision MCQs ‘which 
might help them with their revision for their end of 
year exams’ ( ie. an optional aid to help with their 
revision) in the summer of 2004. We wrote our 
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MCQs for a double module course called 
Computer Systems and Networks. The major 
aims of Computer Systems and Networks are to 
enable students to appreciate the features of a 
selection of general purpose computers, computer 
peripherals and network technologies. The 
module also aims to enable the students to 
acquire the technical basis to understand semi- 
technical evaluations of computer systems and to 


interpret manufacturers’ publicity about their 
systems. The final aim of the module is for 
students to appreciate how hardware and 
software support High Level Languages. 
Computer Systems and Networks is therefore a 
highly technical subject and has a reputation 
among students for being a difficult subject, which 
thus rendered it ideal for our trial! 



CS&N MCQ 

Figure 1: The proportion of workers:lurkers per Computer Systems and Networks (CS&N) MCQ. 


In this initial study, we made the revision MCQs 
available to a cohort of 214 students. Figure 1 
shows the ratio of workers: lurkers for each of our 
50 revision MCQs. There were 93 students who 
initially logged onto our revision on-line learning 
system, approximately 43% of those who could 
have logged on. Only 21 (approximately 10%) of 
the students looked at all 50 MCQs. This means 
that there were 121 (approximately 57%) shirkers 
who did not use our revision on-line learning 
system at all; this deteriorated to approximately 
90% by the end of the revision MCQs. Of the 93 
students who looked at the first MCQ, 76 
(approximately 83%) were workers and 17 
(approximately 17%) were lurkers. By the time the 
students had experienced 7 MCQs the number of 
workers/lurkers tailed off to about 35 
(approximately 16%) students, which continued to 
decline to 24 (11%). The number of 
workers/lurkers remained around 24 for the rest of 
the 50 MCQs. 


Unknown to our students we ranked each MCQ 
into one of three categories easy, moderate or 
hard. We considered 25 MCQs to be easy, 16 
MCQs to be moderate and 9 MCQs to be hard. 
We also estimated that over 60% of all of our 
students would be able to successfully answer an 
easy MCQ within 60 seconds. We estimated that 
about 50% of our students would be able to 
answer a moderate MCQ successfully in under 
120 seconds. Finally we estimated that less than 
50% of our students would be able to answer a 
hard MCQ successfully in less than 180 seconds. 

Overall the percentage of worker students 
answering MCQs correctly was in the range 41% 
to 92%. Figure 2 shows the percentage of workers 
answering each of our three categories of MCQs 
successfully. Our estimation for the proportion of 
workers successfully answering each category of 
MCQs correlated well with the actual percentages 
from the workers’ responses. Approximately 67% 
of the workers successfully answered the easy 
MCQs, approximately 53% of the workers 
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successfully answered the moderate MCQs and answered the hard MCQs. 

approximately 32% of the workers successfully 
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Figure 

However, our estimates for the amount of time For five MCQs there were no lurkers. Intuitively, 

proved to be incorrect, as shown in Figure 3. We we thought these would be easy MCQs which 

expected the student response times to be far were answered quickly and correctly by all 

greater than they actually were. For example, we students. This turned out to be a false premise, 

expected a worker to successfully respond to an We had categorised three of the five MCQs as 

easy MCQ within 60 seconds; the successful easy and the other two as moderate. Furthermore, 

response time for an easy MCQ turned out to be the range of correct responses was 40 - 75% and 

within 45 seconds. It is clear that some of the the response time was 16 - 43 seconds, 

lurkers considered responding because the lurker 

time was within 53 seconds. For the moderate We identified a small number of MCQs that 

MCQs, the worker’s successful response time caused students some problems. This was 

was less than that of the easy MCQ response because: few students answered an MCQ 

time and was within 39 seconds. The lurker correctly, the time taken to respond to answer an 

response time was within 50 seconds. Finally, the MCQ was either greater than or less than our 

successful worker response time was within 29 expected time. In the case where the time was 

seconds for the hard MCQs and the lurker less than our expected time and the proportion of 

response time was within 42 seconds. students answering the MCQ correctly was high 

then this meant that the MCQ was probably not 
Our results suggest that students appraise MCQs sufficiently challenging. In the case where the 

quickly, if they consider that they can answer the student response time was far less than our 

MCQ they respond, if they consider that they expectation and the proportion of students 

cannot answer the MCQ they may think about it answering the MCQ correctly was low, then this 

for a short period of time and then request the MCQ was either too challenging for the students 

answer. We also consider that students form a or we may have taught the material not as clearly 

decision quickly and if they consider the MCQ to as we should, 

be too difficult for them to answer they either 
guess an answer or request the answer and 
become a lurker for that MCQ. 



Easy Moderate Hard 


2: The percentage of students correctly answering each category of MCQs. 
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3.2 Study 2 

We decided to re-evaluate our results in a second 
study. This second study was offered to a new 
cohort of students in the first semester of this 
academic year (2004/2005). Again, we invited first 
year Computer Science undergraduates for the 
same double module course, Computer Systems 
and Networks, to participate and we also included 
MCQs from a second double module course 
entitled Formal Systems. 

The aims of the Formal Systems module is to 
enable students to develop confidence in formal 
and mathematical modes of discourse, to 
experience a range of formalisms that are useful 
in the design of programmed systems and for the 
students to appreciate the relationships between 
mathematics, formal (symbolic) reasoning and 
programmed systems. Similarly to Computer 
Systems and Networks this course has a student 
reputation for being a difficult subject, which 
renders it ideal for our study. 

In our second study, we provided 10 formative 
revision MCQs for Computer Systems and 
Networks and a further 10 formative MCQs for 
Formal Systems which, as in the first study, might 
help the students with their revision for their (in 
this case) mid-semester summative assessments. 
There were a number of key differences between 
this study and the first study. First, we could not 
offer the same cohort of students MCQs because 


they would now be in their second year of study. 
In the second year of study, MCQs are not 
generally used to assess our undergraduate 
students. Instead our second year students 
undertake a number of courseworks assigned at 
different times throughout the academic year, 
followed by traditional written exams at the end of 
the second semester. Therefore, this type of study 
would provide little benefit to these students. 
Consequently, we decided to continue our study 
with first year undergraduates, even though it 
meant it was a different cohort. The second major 
difference was that we offered the formative 
MCQs from two different courses. Third, we 
reduced the number of MCQs to 10 for each 
course and fourth, the MCQs were provided as 2 
banks of 5 MCQs for each course. We 
deliberately reduced the number of MCQs 
because of the rapid reduction in those attempting 
the MCQs in the first study and because this new 
cohort of students had only studied for a short 
period of time. We provided the MCQs as two 
banks of 5 because we considered students 
would be more likely to complete the MCQs if they 
were in short sequences which they could 
undertake all at once or even by attempting the 
MCQs on two different occasions. We offered two 
different courses to investigate if there would be 
different responses to different courses. A total of 
155 students undertook the mid-semester 
summative tests. 
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In study 2, Figure 4 reflects Figure 1 of study 1, 
except in this study there were only 20 MCQs 
which were divided into two banks of 5 MCQs for 
Computer Systems and Networks and a further 
two banks of 5 MCQs for Formal Systems. More 
students attempted the Formal Systems MCQs 
than the Computer Systems and Networks MCQs. 
We attribute this to one major difference; in both 
cases the students were e-mailed to inform them 
of the revision MCQs, however the Formal 
Systems tutors, also, verbally informed their 
students about the revision MCQs during class 
contact time, whereas the Computer Systems and 
Network tutors did not. 

Initially 55 students logged onto the first bank of 
Computer Systems and Network revision MCQs, 
approximately 36% of those who took the 
summative MCQ a short while later. By the end of 
the first bank of MCQs this number had reduced 
to 36 (approximately 23% of those that took the 


summative MCQ). 39 (approximately 25% of 
those that took the summative MCQ) students 
logged onto the second bank of Computer 
Systems and Networks MCQs which declined to 
26 (approximately 17% of those that took the 
summative MCQ) by the end of the second bank 
of MCQs. The number of logons for the Formal 
Systems MCQs was more encouraging; 138 
(approximately 89% of those that took the 
summative MCQ) of the students initially logged 
onto the first bank of MCQs. This declined to 83 
(approximately 53% of those that took the 
summative MCQ) by the end of the first bank of 
Formal Systems MCQs and 78 (approximately 
50% of those that took the summative MCQ) of 
the students logged onto the second bank of 
Formal Systems MCQs, which declined to 58 
(approximately 37% of those that took the 
summative MCQ) by the end of the second bank 
of MCQs. 



1 2345 1 2345 1 23451 2345 

Bank 1 CS&N Bank 2 CS&N Bank 1 FS Bank 2 FS 

Figure 4: The proportion of workers:lurkers per Computer Systems and Networks (CS&N) MCQ, and per 
Formal Systems (FS) MCQ. 


Of the 55 students who initially logged onto the 
first bank of Computer Systems and Network 
MCQs, approximately 84% were workers and only 
approximately 16% lurkers. By the end of the first 
bank of MCQs the proportion of workers remained 
relatively high at approximately 80% and this was 
mirrored with the second bank of MCQs. Similarly, 
of the 138 students who initially logged onto the 
first bank of Formal Systems MCQs, 
approximately 95% were workers and only 


approximately 5% were lurkers, which declined to 
about approximately 85% of workers at the end of 
the first bank of Formal Systems MCQs. Again, 
this was mirrored in the second bank of Formal 
Systems MCQs. 

As with Study 1, in this second study we ranked 
our MCQs into the same categories (easy, 
moderate and hard). We also applied the same 
time estimation to each category (an easy MCQ 
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should be answered successfully by 60% of our Figure 5 shows the percentage of workers 

students successfully within 60 seconds, a answering each of our three categories of MCQs 

moderate MCQ should be answered successfully successfully for both courses. Again, our 

by 50% of our students within 120 seconds and a estimation for the proportion of workers 

hard MCQ should be answered successfully by successfully answering each category of MCQs 

50% of our students within 180 seconds), we correlated reasonably well with the actual 

applied these metrics to both Computer Systems percentages from the workers’ responses, 

and Networks, and Formal Systems. We Approximately 70% of the workers successfully 

categorised 4 of the Computer Systems and answered the easy Computer Systems and 

Networks MCQs as easy, 4 as moderate and 2 as Network MCQs, approximately 48% of the 

hard, similarly we categorised 3 Formal Systems workers successfully answered the moderate 

MCQs as easy, 4 as moderate and 3 as hard. Computer Systems and Network MCQs and 


approximately 65% of the workers successfully 
answered the hard Computer Systems and 
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Figure 5: The percentage of students correctly answering each category of MCQs 


In study 1, our estimates for the amount of time 
proved to be incorrect and we changed our 
Computer Systems and Network MCQs in study 2 
to suit our time estimations. We decided to 
change the MCQs rather than our time 
estimations, because in summative assessment 
students have a strict time limit and therefore we 
did not want to increase or decrease student 
response time. In study 2, the student response 
time for those students who correctly answered 
our Computer Systems and Networks easy 
questions was within 114 seconds, approximately 
double our time estimation, as shown in 


160 

140 

120 


«T 100 
o 



CS&N Easy CS&N CS&N Hard F 

Moderate 

Figure 6: Worker/lurker response time for each 
category of MCQ for Computer Systems and 
Networks (CS&N) and Formal Systems (FS). 

. However, our time estimations for the moderate 
and hard MCQs closely mapped to the student 
response times (within 127 seconds for the 
moderate MCQs and within 143 seconds for the 
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hard MCQs). The successful response time 
estimations for the Formal Systems MCQs proved 
to be far greater than the actual student response 
time, the easy Formal Systems MCQs were 
correctly answered within 33 seconds, the 
moderate MCQs within 81 seconds and the hard 


MCQs within 59 seconds. These results verify our 
premise from study 1 that students appraise 
MCQs quickly and respond by either submitting 
an answer or requesting the solution. 



CS&N Easy CS&N CS&N Hard FS Easy FS Moderate FS Hard 

Moderate 


Figure 6: Worker/lurker response time for each category of MCQ for Computer Systems and Networks 
(CS&N) and Formal Systems (FS). 


4. Conclusions and discussion of 
future work 

Our results have helped us to answer the three 
questions that we were attempting to answer. 

How does the tutor know the proportion of 
students who are actually undertaking the 
formative exercises? 

How does the tutor know if a student is just 
looking at the solutions and not doing the 
formative exercises? 

How does a tutor know how long a student is 
spending on each formative exercise? 

Using our on-line learning system we can 
estimate the maximum number of students 
attempting the formative MCQs. It was interesting 
to note that more of the Formal Systems students 
engaged in our on-line learning system than the 
Computer Systems students, yet they were from 
the same cohort studying the same subjects. The 
Formal Systems tutors verbally encouraged their 
students to attempt our on-line revision MCQs. 
This indicates that on-line learning is not a 
substitute for student/tutor contact but should be 


used in a blended manner to augment each 
teaching strategy and to improve the student 
learning experience. 

Using our on-line learning system, we can now 
suggest a classification of students into three 
categories: workers, lurkers and shirkers. We are 
disappointed with the high number of shirkers and 
are now actively considering ways to encourage 
more students to engage in such blended 
teaching/learning systems. 

We can now estimate the amount of time students 
spend on formative questions. In our case, this 
helped us to change our expectations between 
the first and second study. Since students are 
often placed under time constraints during 
summative assessments, we consider that we as 
academics must provide the students with a fair 
and reasonable amount of time to answer our 
summative questions. Our on-line learning system 
is providing us with good indicators of how this 
might be achieved and is making us re-evaluate 
our question timing schedules. Classifying 
questions into different categories is helpful 
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because we can associate each category with a 
different time metric. It is also in line with similar 
studies on using MCQs effectively such as the 
work undertaken at University of Leicester and the 
guidelines of the University of Cape 
Town(University of Leicester, 2005) 

We would like to identify which student is using 
our on-line learning system and when they are 
using it. Currently at the University of 
Hertfordshire StudyNet does not provide us with 
the ability to identify individual students. When we 
are able to identify individual students, we plan to 
correlate the use of our on-line learning system 
with class attendance and to compare results of 
summative assessments from results obtained 
from of our formative assessments. 

From this proposed new study we would be able 
to identify in more detail how students engage 
with their studies. First, we would be able to 
quantify the number of occasions and the amount 
of time each student engages with our formative 
MCQs. Currently, we cannot quantify usage in an 
absolute manner. We consider that it would be 
beneficial for us as academics to be able to 
correlate the student usage frequency and student 
frequency usage time that is spent on formative 
MCQs. We could then use this information to 
identify areas of student concern with the taught 
material, and then we could dynamically amend 
our teaching material to improve the student 
learning experience. Second, we would be able to 
compare both individual student and cohort 
summative assessment results with on-line MCQ 
formative assessment results. One consideration 
is to repeat a previously used summative MCQ 
assessment as a formative MCQ assessment but 
to a different cohort of students. We could then 
estimate with a higher degree of accuracy the 
actual amount of time a student spends on each 
MCQ without the additional pressure of a timed 
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